Enhancements to data-driven media management within an electronic device

ABSTRACT

A centralized resource manager manages the routing of audio or visual information within a device, including a handheld device such as a smartphone. The resource manager evaluates data-driven policies to determine how to route audio or visual information to or from various input or output components connected to the device, including headphones, built-in speakers, microphones, bluetooth headsets, cameras, and so on. Among the data considered in the policies are connection status data, indicating if a device is connected, routing status data, indicating if a device is permitted to route information to or from a component, and grouping data, indicating logical relationships between various components. Components may be considered inherently routable, automatically routable, or optionally routable. Numerous other uses exist for such data, including providing simpler and more logical management interfaces.

PRIORITY CLAIM

This application claims benefit under 35 U.S.C. §120 as aContinuation-In-Part of: U.S. patent application Ser. Nos. 11/824,160filed on Jun. 28, 2007, now U.S. Pat. No. 7,861,008 and 11/824,320,filed Jun. 28, 2007, and U.S. patent application Ser. No. 12/106,187,filed Apr. 18, 2008. The entire contents of each of which are herebyincorporated by reference as if fully set forth herein. The applicant(s)hereby rescind any disclaimer of claim scope in the parentapplication(s) or the prosecution history thereof and advise the USPTOthat the claims in this application may be broader than any claim in theparent application(s).

FIELD OF THE INVENTION

The present invention relates to routing audio within an electronicdevice using a variety of sources and outputs.

BACKGROUND

The approaches described in this section are approaches that could bepursued, but not necessarily approaches that have been previouslyconceived or pursued. Therefore, unless otherwise indicated, it shouldnot be assumed that any of the approaches described in this sectionqualify as prior art merely by virtue of their inclusion in thissection.

Computing devices often process multiple types of media. The media thatmay be processed by computing devices often include numerous forms andformats of video, and numerous forms and formats of audio. Within suchcomputing devices, many applications may be competing for media input orplayback. Examples of the types of applications that may attempt to playor input media within a handheld computing device include, for example,a telephone application, a web browser, an e-mail application, a shortmessage service (SMS), a music player, a video player, instant messaging(IM) application, a multimedia message service (MMS), a voice over IP(VOIP) application, and a teleconferencing application.

Within computing devices, the various software applications that utilizemedia all contend for use of the various input or output resources, andhave different rules and protocols for doing so. Some of these rules andprotocols may conflict from one application to another.

To complicate matters further, many computing devices also have multiplepossible inputs and outputs for routing audiovisual information. Forexample, a handheld computing device may have a line-out port, aheadphone port, a docking port, a microphone, a line-in port, a wirelessconnection, and multiple built-in speakers, each of which may serve adifferent function and have different capabilities.

One way to handle the contention among the applications for the playbackresources of a computing device is to design each application with logicto check the state of settings of the device and decide for itselfwhere, when, and how to play its audiovisual information. Unfortunately,using this approach, each software application developer must implementcode for a behavior tree that attempts to take into account the varietyof factors that can impact media processing on a device. The result maybe less-than-optimal media handling decisions made bymore-complex-than-necessary applications.

Further complicating the routing of audiovisual information, it may bedesirable to have the information processed in specific ways underspecific circumstances. For example, when music and voice audio are tobe played concurrently, it may be desirable to mix the music and voiceaudio in a particular way. Typically, such custom treatment of specificcases would be hard-coded into the logic of the software and/or hardwarecomponents involved in the playback. Hard-coding such customizationssignificantly increases the complexity of a devices software and/orhardware components.

Additional complications arise when a device is connected to one or morewireless inputs or outputs, such as bluetooth headsets. While manyconventional devices simply assume that audiovisual information shouldbe routed through wireless connections whenever possible, a user mayoften prefer not to route audio through a connected device. For example,the user may have inadvertently entered into the range of such a device.Many conventional devices would automatically assume that audio shouldbe transferred to the wireless device, even though the user may in factnot be in a position to even hear the device.

Further complications arise in managing certain accessory devices thatinclude several logical inputs or outputs—such as Bluetooth headsetsthat implement multiple audio protocols—or in managing groups of inputsand outputs that have complementary uses—such as a headset and amicrophone. Users and software developers alike find management of suchgroups of inputs and outputs confusing and tedious.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention is illustrated by way of example, and not by wayof limitation, in the figures of the accompanying drawings and in whichlike reference numerals refer to similar elements and in which:

FIG. 1 is a block diagram of a handheld device upon which embodiments ofthe present invention can be located;

FIG. 2 is a functional overview of an embodiment of the presentinvention;

FIGS. 3A and 3B are example data tables according to an embodiment;

FIG. 4 is a block diagram of an example audio hardware arrangementaccording to an embodiment;

FIG. 5 shows examples of nodes, connections, routes, and properties,according to an embodiment;

FIG. 6 is a block diagram of an example audio hardware arrangementaccording to an embodiment;

FIG. 7 shows an example of cycling through data tables according to anembodiment;

FIG. 8 depicts an example table for storing policy and status data,according to an embodiment;

FIG. 9 is an example process flow for routing audiovisual information inan electronic device using routing status, according to an embodiment;

FIG. 10 is an example process flow for handling routing status forresources that connect to an electronic device, according to anembodiment;

FIG. 11 depicts an example table for storing policy and grouping data,according to an embodiment;

FIG. 12 is an example process flow for handling routing status forresources that connect to an electronic device, according to anembodiment; and

FIG. 13 is block diagram of a computer system upon which embodiments ofthe invention may be implemented.

DETAILED DESCRIPTION

In the following description, for the purposes of explanation, numerousspecific details are set forth in order to provide a thoroughunderstanding of the present invention. It will be apparent, however,that the present invention may be practiced without these specificdetails. In other instances, well-known structures and devices are shownin block diagram form in order to avoid unnecessarily obscuring thepresent invention.

Embodiments are described herein according to the following outline:

-   -   1.0. General Overview    -   2.0. Structural Overview        -   2.1. Example Media Device        -   2.2. Example System for Managing Audio    -   3.0. Functional Overview        -   3.1. Categories        -   3.2. Policies        -   3.3. Media Server Module        -   3.4. Policy Module        -   3.5. Routes        -   3.6. Routing Manager        -   3.7. Ports, Connections, Nodes, and Routes        -   3.8. Routing Status Data        -   3.9. Grouping Components    -   4.0. Implementation Examples        -   4.1. Example Audio Routing Infrastructure        -   4.2. Properties        -   4.3. Aliasing        -   4.4. Fall Back    -   5.0. Implementation Mechanism—Hardware Overview    -   6.0. Extensions and Alternatives

1.0. General Overview

Approaches, techniques, and mechanisms are disclosed for routingaudiovisual information through an electronic device, including ahandheld computing device such as a mobile phone.

According to an embodiment, audio and/or video is routed within a deviceto or from various input or output components based on policy dataindicating one or more routing policies. Policies are rules for making avariety of determinations useful for identifying a route for audiovisualinformation. Policies are conditioned upon some type of informationmaintained by the device, including status data, device or componentproperties, user preferences, and other policy data. The policies maytake into account, for example, categories associated with requests toinput or output audiovisual information, properties of various input oroutput resources, user preferences, and system or application statusinformation (including information indicating which external audiovisualcomponents, such as headphones, microphones, bluetooth devices, havebeen connected to the ports of the device). Routing of audiovisualinformation for the entire device may be controlled from one or morecentralized modules, hereinafter referred to collectively as a resourcemanager. Accordingly, software applications themselves are not requiredto implement logic for controlling input and output resources. Moreover,the policies along with the requisite data upon which the policies areconditioned may be maintained in a database within the device at alocation accessible to resource manager. Thus, the actual code of theresource manager does not have to be modified to change routing rules.

According to an embodiment, instead of just considering whether an inputor output component is connected to the device, a resource manager mayalso base policy decisions upon routing status data indicating whetheror not a connected component is “routable”—i.e. whether or not thedevice is permitted to route audio or video to or from a connectedcomponent. Some components, such as internal components that are alwaysconnected (e.g. a built-in speaker) may be considered inherentlyroutable. Inherently routable components are components whose connectionstatus never changes, and consequently are assumed to be routable unlessotherwise specified. Some components, such as external components thatrequire a user to physically perform some action to initiate connection(e.g. wired headphones), may be considered automatically routable.Automatically routable components are components whose connection statusimplies the user's preference towards routability, because the userwould not have connected the component if the user did not intend thecomponent to be routable. Automatically-routable components may be setto routable automatically in response to connection. Other components,such as external components that may potentially connect without useractivity (e.g. wireless headphones), may be consideredoptionally-routable. “Optionally-routable components” are componentswhose connection status changes, but without necessarily implying theuser's preference for routability. Unlike automatically routablecomponents, the connection of an optionally-routable component is notassumed to, in and of itself, indicate that the component is routable;rather the component typically only becomes routable after the devicereceives an input (e.g. a user “gesture”) indicating that the device mayroute audiovisual information to or from the component. In someembodiments, a component may be designated as inherently, automatically,or optionally routable based upon policies. The policies may beconditioned upon data such as user or vendor preferences.

According to an embodiment, policies may be designed to treatconnected-and-routable components differently thanconnected-but-not-routable components when determining how to routeaudiovisual information. For example, a policy may cause a device toactually route audio only to connected-and-routable devices, even thoughthe device may display connected-but-not-routable components to the useras alternative targets for the audio. If the user chooses to routeaudiovisual information to or from a connect-but-not-routable component,the routing status for the component may be set to routable. Thecomponent may remain routable for a predefined duration, or for aduration that is dictated by certain policies. For example, thecomponent may remain “routable” only for the duration of the currentrouting operation, for a duration lasting until the component isdisconnected (e.g. the device or the component is turned off, or thecomponent is out of wireless range), or for a duration lasting until theuser specifies that the component is no longer routable.

According to an embodiment, a resource manager may maintain “grouping”data indicating logical relationships between components. For example, a“sibling” relationship may be established between components that havecomplementary uses, such as the microphone and earphone of a singleheadset. Similarly, a “partner” relationship may be established betweencomponents of a same accessory device having mutually exclusive uses,such as a headset profile of a bluetooth device and an A2DP profile ofthat bluetooth device. These and other relationships may be consideredin policies when making routing decisions. For example, the earphone ofa headset may be made routable in response to the microphone of the sameheadset being made routable. Other interactions with the components mayalso occur on a relationship basis (as opposed to acomponent-by-component basis).

As shall be described hereafter, the routing techniques described hereinmay be implemented on a computing device, such as a desktop computer, alaptop computer, a PDA, a phone, or any other device through which audioor video data is routed. Further, a computer-readable storage medium,such as memory, a magnetic disk, etc, may be encoded with instructionswhich, when executed, cause the techniques to be performed.

As used herein, the terms “audiovisual information,” refers to any typeof data stream comprising audio and/or visual information forpresentation to a user or input to an application. The terms “component”or “resource” refer to any logical component to or from which a devicemay route audiovisual information, including internal components of thedevice, external components that are connected to the device by a cableor some other physical means, and external components that are connectedto the device wirelessly. Components may be standalone accessory devicesor one of any number of software or hardware elements of the accessorydevice. For example, a component may be, without limitation, a built-inspeaker, a built-in microphone, an external speaker, an externalmicrophone, a headset, headphones, a wireless device, an instantiationof a bluetooth profile, a touch-screen display, an LED, a monitor,hardware capable of vibrating a device, a camera, and so on.

As used herein, the phrase “determining how to route audiovisualinformation through a device” and like refer to the process ofdetermining one or more logical or physical elements of the device to orfrom which the device or software running thereon may send or receive astream of audiovisual information. The techniques described herein areapplicable to routing audiovisual information for both output to acomponent (e.g. an audio or visual presentation of information) andinput from a component (e.g. audio for a telephone call or video for arecording).

2.0. Structural Overview

The techniques described herein for handling the input and/or output ofaudiovisual information within a device may be applied to any type ofelectronic device that inputs or outputs audiovisual information.Handheld electronic devices, such as personal digital assistants andmobile phones, are merely examples of devices to which the techniquesmay be applied. However, for the purpose of explanation, examples of thetechniques shall be given with reference to handheld devices.

For convenience, the techniques of this application are often describedwith specific reference to the routing of audio within a device.However, the techniques of this application are just as applicable tothe routing of other types of audiovisual information through a device,including, for example, video streams. Thus, for example, techniquesdescribed as being performed with respect to “audio” may also be appliedto “video.” Moreover, for convenience, the techniques of thisapplication are sometimes described in the specific context of audiooutput. However, the techniques are likewise applicable to audiovisualinput of any kind.

2.1. Example Media Device

FIG. 1 shows an example handheld device 100 upon which an embodiment ofthe present invention may be implemented. It is to be noted that FIG. 1is but an example, so that devices other than that shown in FIG. 1 arealso contemplated within the spirit and scope of the present invention.As shown in FIG. 1, the device 100 comprises at least a display 180(which may or may not be touch-sensitive), a main speaker 104, atelephone receiver/speaker 108, a jack 112 for either headphones (listenonly) or headset (listen and talk), a wireless transmitter/receiver 116such as but not limited to Bluetooth®, a line out port 120 (suitable fora docking station used by e.g. larger home-audio system), and also auser interface (UI) speaker 124 (e.g. for keypad click-tones). Device100 may include other types of speakers and audio ports. Device 100 alsoneed not necessarily include all of the features depicted in FIG. 1. Asstated, the device 100 is but one possible embodiment, so that thetechniques described herein are not limited exclusively thereto.

The built-in telephone receiver/speaker 108 may be, for example, similarto a receiver used within a conventional telephone or capable ofoutputting high-fidelity sound. The touch-sensitive version of thedisplay 180 may, for example, contain a keypad which in turn cangenerate sounds when touched.

The device 100 further comprises a mute switch 132, and a vibrationmechanism 136. A mechanism for enabling/disabling the vibration means136 is available through the display 180 by accessing a softwareapplication loaded on the device 100.

Software applications that may be executed by one or more processors onthe device 100 include but are not limited to e-mail, telephone,voice-mail, web-browser, short messaging service (SMS), entertainmentplayer either for music or video, camera, and slideshow (with musicaccompaniment).

The UI speaker 124 is intended mainly for playing keypad sounds (clicks)and notifying the user that certain steps and key-actions may beillogical or not permissible (beeps). However, under certaincircumstances this policy can be overridden.

2.2. Example System for Managing Audio

FIG. 2 shows an example system 200 for managing audio routing within anelectronic device, according to one embodiment of the present invention.FIG. 2 is but one example of a possible system, and other types ofarrangements could also be achieved within the spirit and scope of thepresent invention. Within the system 200, a media server 204 acts as aninterface and coordinator with the various software applications thatmay need to use audio resources within the device 100.

The media server 204 achieves this partly by communicating with an audiopolicy manager 208 in order to determine what resources are available,and what policies exist for playing audio. The audio policy managerstores various policies and other resources in tables 210 _(1-n). Theaudio routing manager 212 is responsible for determining a suitableroute for the requested audio.

The system 200 also tracks what has happened in recent user history. Notjust the most recent user choice (“last in”), but also second to lastin, third to last in, and potentially further back.

System 200, and any other systems capable of managing the flow ofaudiovisual information through an electronic device, is hereinafterreferred to collectively as “a resource manager.” Moreover, actionsreferred to as being performed by system 200 (or any other resourcemanager) may be performed by any component of system 200. Likewise,applications performed by one component of system 200 may instead beperformed by other components of system 200. System 200 may furtherinclude additional components, as needed, for the purposes of routingand determining how to route requests to use input or output resources.

In some embodiments, the components of system 200 may be more broadlyused for purposes other than audio input or output. For example, audiorouting manager 212 may be a more general routing manager for routingpicture or video inputs and outputs. Audio policy manager 208 may be amore general policy manager storing policies for playback or input ofall types of audiovisual information.

3.0. Functional Overview 3.1. Categories

Within various conventional devices, each software application mustimplement its own decision tree for routing audio by itself. Since thelogic for selecting audio resources is implemented differently fromsoftware application to software application, the user is often unableto anticipate how a given software application will utilize audioresources. Moreover, software developers must devote time and resourcesto developing their own decision trees for managing sound. By contrast,according to an embodiment, decisions for routing audiovisualinformation within a device may be centralized within a system such assystem 200 by utilizing information about sound categories that isforwarded from software applications to, for example, the media server204.

To this end, software applications request the use of input or outputresources within the device 100 by specifying categories of audio to themedia server 204. These categories are established for audiovisualinformation based on the intent of the audiovisual information (which isdifferent from the format and/or source of the audiovisual information).Examples of potential audio categories within the system 200 include butare not limited to music, video soundtrack, voice telephone call,telephone voice mail, e-mail notification, short message service (SMS)notification, user interface sounds, ring tones, and alerts. UI soundscomprise separate sub-categories of audio, including but not limited toclicks and beeps.

3.2. Policies

General factors used to determine how to route audio can include, forexample, the audio category, the status of a switch such as physicalswitch 132, the currently connected input or output components (e.g.headphones, wireless), routability data indicating input or outputcomponents that the device is permitted to utilize, grouping dataindicating logical relationships between input or output components,user preferences, and the current state of the device 100 (e.g. powerstatus, telephone call status, media player status, the time of day, andstatus data provided by other applications or resources such as acalendar or GPS device). Additional factors may include, for example,the identity of the software application that wishes to route the audio.This and other information assists in forming the audio policies thatare used by the system 200.

According to an embodiment, a user interface for audio policies withinthe device 100 may be designed simplistically so as to avoid introducingtoo many choices to the user and also render the meaning of the policiesclear to the user, so that the user is never surprised or left with thequestion “why did that happen?” Accordingly, a set of default policiesexist per audiovisual information category. For example, one suchdefault may be that user-initiated sounds (such as the UI keyclicksound) are muted when the mute switch 132 is enabled. Thus, even thoughthese sounds are expected by the user and not very obtrusive, if themute switch 132 is enabled, the key clicks are not heard. Such defaultpolicies may, however, be changed.

According to an embodiment, policies may also prohibit certain types ofaudio routing. Again, these policies can be over-ridden. For example,one policy may be that music is unlikely to be played to the UI speaker124. Yet another (waiveable) policy may be to not play music through thewireless speaker 116, although in certain cases music could be mixed in.These policies are not fixed and immutable, but instead part of theflexibility of matching a user interface with the hardware capabilitiesof a device.

As another example of a policy, if a telephone call is in progress, apolicy may assume that only one audio route should exist, and thus routeall audio through the same speaker. This is because the user may befocused on the telephone call, and potentially unable to hear any of theother audio speakers regardless of whether those speakers are availableor not. Accordingly, if an e-mail notification comes in, the policy mayindicate that device 100 should mix the e-mail notification into thetelephone audio.

As yet another example of a policy, if a notification is deemed to besufficiently important (by policy), that notification may be broadcastacross all speakers.

3.3. Media Server Module

The media server 204 is a centralized mechanism for handling audioassociated with the various media-playing software applications.Accordingly, the various software applications do not need to have theirown logic and resources for deciding how to manage audio. Instead, thesesoftware applications merely forward a request indicating their intentto route a certain type of audio to the media server 204. As part of therequest, the software applications disclose a specific audio category tothe media server 204. The software applications then depend on the mediaserver 204 to consider policies and factors such as those describedherein, so as to ensure the appropriate audio gets properly routed.Where possible, the media server 204 will honor the intent forwarded byan application, but in some cases may be forced to refuse a request oremploy a different strategy than requested by the application.

The types of decisions expected of the media server 204 include but arenot limited to determining: whether to route the audiovisual informationat all, through which port to input or output the audiovisualinformation, a volume level at which to input or output audio, aspecific data stream to play, whether to vibrate, whether to click,whether to blink the display 180, as well as other tasks. The mediaserver 204 evaluates the requests forwarded by the various softwareapplications partly by classifying their various audio requirements intoaudio categories.

The media server 204 centralizes and ensures consistent audio decisionsthat do not surprise the user. The media server 204 discerns the intentof the audio category, and then communicates the requested category tothe audio routing policy module 208, which references the various datadriven tables 210 _(1-n). Whenever the user adds or removes a piece ofhardware, the data tables 210 _(1-n) are updated. Accordingly, the audiorouting policy module 208 always knows what is connected to the device100. The audio routing manager 212 then tells the device 100 how toroute the audio.

3.4. Policy Module

The audio policy module 208 enforces policies for each of the numerousaudio categories. The audio policy module 208 can use metadata-basedrepresentations of these policies within the tables 210 _(1-n), asopposed to the policies being coded into executable logic. However, thepolicies utilized by audio policy module 208 may be code driven instead.

As shown in FIG. 2, alongside both the media server 204 and the audiorouting policy module 208, several data tables 210 _(1-n) are used,thereby forming a multi-dimensional decision matrix. As the user changestheir preferences and takes other actions that have input or outputresource implications, the system 200 updates various of these datatables 210 _(1-n), while others may remain unchanged. System 200 issufficiently intelligent to distinguish between the user connected aheadphone components as opposed to headset components to port 112, andto update various of the tables 210 _(1-n) accordingly.

Last-In Wins Policy

One example policy that may be implemented by the audio policy manager208 is a “last in wins policy.” According to the “last in wins policy”,in situations where multiple connected devices are capable of servingthe same role in a routing operation, the most recently connected deviceis selected to play that role. For example, assume that a user plugs inheadphones 112 after a wireless connection 116 to a headset has beenestablished. Under these conditions, both the wired headphones and thewireless headset may be capable of playing audio from an incoming call.Based on the last in wins policy, the headphones 112 are selected forplaying the audio of calls based on the assumption that the user pluggedin the headphones 112 because the user wants to hear audio through theheadphones 112, and not the wireless 116. Thus, system 200 may employthe “last in wins” policy. In essence, the “last in wins” policycodifies the assumption that the last action the user took to affect adevice's input or output resources indicates how the user wants toexperience audiovisual information.

Policy Overrides

However, like any policy, it may be desirable to override the “last inwins” policy in certain circumstances. To this end, the policies storedor expressed in the tables 210 _(1-n) are not rigidly enforced, butinstead serve as a guide to the media server 204 and the audio policymodule 208. There can therefore be user-initiated overrides of thepolicies. In carrying out its responsibilities, the audio policy module208 may consider various other policies in data tables 210 _(1-n). Forexample, another policy may indicate that audiovisual informationdesignated as being in an anti-“wireless” category may not be routedthrough a wireless device. For instance, audio in this category may benot suitable for playing over wireless (e.g. UI sounds). If the“last-in” audio component supports wireless, but the category is“anti-wireless—then the audio policy module 208 will instead select that“last-in” “non-wireless” audio output device (e.g. UI speaker 124). Thisis a simple example where the “last-in wins” policy is conditional andnot always obeyed.

Another example of an override is a user-initiated override. Forexample, a software button on the display 180 may allow a user to diverta telephone call to the main speaker 104. This clear and unmistakableuser choice may override other policies, including even the “last inwins” policy.

As another example, suppose the mute switch 132 is enabled. One policythat may be implemented while mute switch 132 is enabled is thatuser-initiated sounds (e.g. typing feedback) may still sound—that is, benon-muted—but that non-user-initiated sounds (e.g. e-mail notificationsor SMS notifications) do not sound. Setting aside the mute switch 132,another policy may be that user-initiated clicks such as typing soundsand other beeps are generally not routed to the main speaker 104.

Yet another example policy is that the wireless 116 and telephonereceiver 108 outputs are unlikely preferences when the audio category ismusic, and thus should appear as extremely low in priority within thetables 210 _(1-n), if at all.

According to one embodiment, within the system 200 there are certainpolicies that are unaffected by the user actions. Other policies inother tables, however, change depending on user-derived actions. Forexample, when a user adds or removes an input or output component,various status or priority data in data tables 210 _(1-n) may change andthus affect policies expressed in those tables.

Consideration of Multiple Policies

Audio policy module 208 may need to consider a variety of differenttypes of policies to determine how to route audiovisual information. Forexample, one of the data tables 210 _(1-n) may hold the policy data:“user prefers to be apprised of incoming e-mail notification.” Thusaudio policy module 208 may determine from this policy data that theuser prefers that the e-mail notification be permitted to sound.However, specific mechanisms for performing notification may beindicated by a different data table. The audio policy module 208 wouldtherefore need to consult other policies in the various data tables 210_(1-n) and decide 1) whether to generate the notification as audiosound, vibration feedback, both audio sound and vibration feedback, orneither; 2) through which audio output to route any audio for thenotification.

Status, Priority, and Resource Data

According to an embodiment, the data tables 210 _(1-n) also containinformation about the properties of routes, connections that belong toroutes, and nodes that belong to connections. The properties describehow the nodes, connections, and routes should be configured. The valuesof such properties may affect a variety of decisions, including whetherroutes are selected, whether specific connections within routes areoptional, and how audio is processed at specific ports when specificconnections are used to route the audio.

According to an embodiment, audio policy module 208 stores dataindicating the order in which pluggable input or output components havecome online within various of the tables 210 _(1-n). Such data may be,for instance, a prioritized list. According to an embodiment, audiopolicy module 208 also stores within the data tables 210 _(1-n) a map ofper-audio-category information including whether that audio category isbroadcast to multiple outputs, whether that category obeys or ignoresthe ringer/mute switch 132, and a prioritized list of routes, such asthose shown in FIG. 6. According to an embodiment, audio policy module208 further stores a list of outputs (both pluggable and non-pluggable)which should be ignored if the ringer/mute switch 132 is enabled.

Events

According to an embodiment, events within device 100 may cause system200 to change audio categories. Examples of such an event include a songending or a user clicking a key on the display 180. At the time of suchan event, the media server 204 may instruct the audio policy module 208to change categories. Audio policy module 208 may then analyze policiesto select a new port through which to route audio—e.g. audio policymodule 208 may review the top of the ordered list of pluggable input oroutput components described earlier.

At the time of such an event, the audio policy module 208 may alsoreview the audio category in the stored map within the data tables 210_(1-n) and determine whether a “last in” pluggable output is stored inthe prioritized list of routes within the data tables 210 _(1-n), suchas those shown in FIG. 7. If the “last in” pluggable output is storedtherein, the audio policy module 208 may select an appropriate routedescription, and requests that the audio routing manager 212 commit tothe route for playback. If the “last in” pluggable output is not storedtherein, the audio policy module 208 may instead read the remainingitems in the map of per audio-category information, and compares thoseitems with the prioritized list of routes. If another output is present,the audio policy module 208 may then select an appropriate routedescription for the other output, and requests that the audio routingmanager 212 commit to the route. Details about the prioritized list ofroutes and how audio routing manager 212 carries out these requests areshown in FIGS. 5-7 and their accompanying descriptions.

According to an embodiment, some events may cause a route change forother reasons. For example, whenever a new pluggable component becomesavailable, the audio policy module 208 may push the pluggable componentto the top of the ordered list of outputs stored within the data tables210 _(1-n), and then looks up the currently applied category in theaudio category map. The audio policy module 208 then determines whetherthe new pluggable component exists in the prioritized list of routedescriptions. If the new pluggable component exists within the existingroute descriptions, the module 208 outputs policy findings to the audiorouting manager 212, and requests that the audio routing manager 212commit the appropriate route.

Example Scenario

As an example scenario, suppose a user wants to listen to music. Theaudio policy module 208 may consult connected status data to determinewhich outputs are connected, routing status data to determine throughwhich of the connected outputs the device is permitted to route audio,as well as device priorities associated with music playback. The audiopolicy module 208 may note that, for example, both headphones as well asthe main speaker are connected and available for playback. The audiopolicy module 208 may then consult a data table holding priorityinformation associated with the music category. The first priority maybe the line out port 120, but that feature is not currently connected.The audio policy module 208 may then proceed down the priority list. Thenext item may be the headphones 112, while main speaker 104 may be thethird item in the priority list. The audio routing policy module 208will accordingly choose headphones 112 for audio output, and mute themain speaker 104. However, other embodiments of the invention may arriveat different routing decisions in this scenario, such as to enable themain speaker 104 simultaneously with the headphones 112.

Example Policy and Status Data

FIG. 3A depicts an example data table 210 ₁ holding some abbreviatedpolicy information, according to an embodiment. According to otherembodiments of the invention, data table 210 ₁ could differsignificantly from that shown in FIG. 3A, including more or lessdetailed information about the status of device connections and thepolicies to be employed by system 200. For purposes of brevity, onlyfive of the audio outlets within the device 100 are shown althoughnumerous others may exist. Similarly, only three types of audiocategories are shown, although again numerous others may exist.

In maintaining the table 210 ₁, the status of various input or outputcomponents is continually queried and updated by the system 200.Consequently, the first column is “present?”, meaning is the specificaudio outlet present, enabled, and available for use. In the exampleshown, the main speaker 104, headphones 112, and UI speaker 124 arepresent, and the asterisk at the headphones 112 means that theheadphones 112 were “last in”. Meanwhile, the wireless 116 and line out120 outputs are not present, and thus not available for use. However,the wireless 116 is shown with two asterisks, symbolizing it was “secondto last in”. As stated, the various tables 210 _(1-n) work together tostore data about user history.

The next three columns show three types of audio categories. The data inthese three columns indicates policies as to whether or not those audioin those categories can be routed to or from the five listed input oroutput components. For clarity, priority information and propertyinformation which may be included in other policy data tables isintentionally not shown in the data table 210 ₁. An example prioritysequence is instead shown in the data table 210 ₂ of FIG. 3B, as shallbe discussed below.

One way for audio policy manager 208 to interpret the policy data indata table 210 ₁ is as follows. Starting with the category “keypadaudio,” the table expresses a general policy that audio in the categorybe played over UI speaker 124, and not through main speaker 104 orheadphones 112. Because the wireless 116 and line out 120 are notconnected, no policy is indicated for those components. As for the “ringtones” category, audio in this category may be played through the mainspeaker 104, and potentially through the headphones 112, but only undercertain policy conditions. To determine what those policy conditionsare, it may be necessary to view data from one of the other tables 210_(2-n). Ring tones should not be played through the UI speaker 124 asgeneral policy. The last category of FIG. 3A is “music,” which may beplayed on the main speaker 104 or the headphones 112, but as generalpolicy is not playable on the UI speaker 124.

Example Priority Data

FIG. 3B shows an example priority data table 210 ₂, according to anembodiment. The example priority data is a subset of the policy datathat may be maintained in the data tables, in that the priority dataexpresses policies for determining which component has the highestpriority for routing. For consistency, FIG. 3B has the same exampleaudio categories as FIG. 3A, with various input and output componentsranked in order of priority for those categories. Again, like with FIG.3A, FIG. 3B is abbreviated, truncated, and simplified for purposes ofbrevity. Other embodiments may include less or more elaborate prioritydata.

One way for audio policy manager 208 to interpret the policy data indata table 210 ₂ is as follows. If a ring tone occurs, the firstpriority would be to play it over the main speaker 104. However, if themain speaker 104 is not available, then the system 200 should play thering tone over the headphones 112. Similarly, if a keypad noise occurs,the audio associated therewith should be played over the UI speaker 124.If music occurs, the first choice should be the headphones 112, but ifthat is not available the second choice the main speaker 104, and ifthose are not available the third would be the wireless 116. Thepresence of the wireless 116 might be stored in the data table 210 ₂despite the fact that the data table 210 ₁ shows that the wireless 116is disconnected. One reason for this presence could be recent usagehistory, specifically that the user recently used the wireless 116 andthus may use it again.

According to an embodiment, in both example data tables 210 ₁ and 210 ₂,the policies and priorities are not absolute, but instead are suggested.Under certain circumstances both policies and priorities may beoverridden.

3.5. Routes

System 200 constructs routes by doing a series of appends of small stepsto form a complete path. In one embodiment, routes are formed by using atype of grammar. This grammar may have varying degrees of complexity. Asimpler embodiment may rely on grammar describing only a source anddestination (i.e. a direct path).

In some embodiments, there may be problems or conflicts that complicatethe route of an audio stream to its intended destinations. Thus, theaudio policy module 208 may rely on a grammar that allows it tointelligently make routing decisions through, for example, a variety ofaudio signal processors. A route in such an embodiment may go beyondsource and destination and also contemplate intermediate points (i.e. anindirect path), thus eliminating some or all of the uncertainties overrouting. Without the intermediate steps, an audio routing manager mustinfer a route only from endpoints (source and destination). It maytherefore be preferable in some embodiments to avoid such inferences.

In other embodiments, the routes are constructed for both direct pathand indirect path situations.

FIG. 4 depicts an example of how the system 200 achieves routing,according to an embodiment. FIG. 4 shows a group of audio processors A,B, C, D, and E. Any or all of these audio processors may have codeccapabilities, and may also have baseband features. FIG. 4 is a blockdiagram version of only a portion of the audio capabilities of thedevice 100, and is simplified for the purposes of brevity.

In FIG. 4 an audio source, such as a software application executing ondevice 100, is shown. For purposes of example, within FIG. 4 it will beassumed that system 200 is responsible for deciding how to route audiofrom the audio source to either the main speaker 104, the headphones112, or both.

An example of a route that would be direct path is:

audio source→main speaker 104.

An example of a route that would be indirect path is:

audio source→audio processor A→audio processor D→main speaker 104.

3.6. Routing Manager

As described in previous sections, audio policy module 208 makes policyfindings by comparing audio category data to data about currentlyattached devices, as well as other criteria including policies andproperties of the ports, nodes, connections, and routes. Then, the audiorouting manager 212 identifies routes from routing data as discussedabove. Finally, the audio routing manager 212 commits an identifiedroute based on the policy findings as well as on the properties storedin the data tables 210 _(1-n). The audio routing manager 212 can achievethis result using a variety of means.

According to an embodiment, the audio routing manager 212 may also servea number of other functions.

Manipulation of an Audio Stream

According to an embodiment, the audio routing manager 212 is responsiblefor implementing policies for manipulating the audio stream, such asmixing audio, removing unwanted pops and other noises, andvolume-matching. One way audio routing manager 212 may achieve suchresults is to ramp up when potential problems or aberrational sounds aresuspected, thereby preventing unwanted pop sounds due to spikes.

As another example, system 200 may support volume selection that onlyaffects certain audio, and not other audio. For example, it is possiblethat a user wishes to only change the volume of an audio stream beingrouted to the headphones 112, but not change the volume of any audiothat may be routed to the main speaker 104. To support such a policy,system 200 reviews the various tables 210 _(1-n), and determines whichof the audio categories is in use at a given time. At that point, theaudio routing manager 212 ensures that the specific category in use isthe only one which the volume selector is linked to. For instance, auser may have set the main speaker 104 to a lower volume, for softermusic, while at the same time have specified a policy to make aring-tone much louder than the music. The audio routing manager 212 maybe responsible for implementing such volume adjustments.

A user may also specify a preference that indicates volume levels thatare inappropriate. A capping policy may be implemented, wherein amaximum volume is imposed under certain circumstances, regardless of auser's selection. The audio routing manager 212 may likewise beresponsible for implementing this volume limitation.

Another policy implemented by audio routing manager 212 may be totemporarily “duck” a first audio stream, such as music, in order totemporarily increase emphasis on a second audio stream, such as a ringtone. When the second audio event has finished, the first audio streammay be restored to its original volume. The system 200 may beresponsible for implementing this “ducking,” as well as for consultingpolicy data to determine which audio categories and under whatcircumstances to “duck” audio.

According to an embodiment, various and perhaps all of the audioprocessors A-E may affect the volume of that audio as heard by the user.Although only five audio processors are shown in FIG. 4, within anyparticular device 100 there may be a large number of audio processors.Consequently, knowing every step from source to destination makes iteasier for audio routing manager 212 to manage all volume levels alongthe route, thereby assuring an appropriate playback experience, with nosudden spikes in volume. All of the elements within FIG. 4 are examplesof nodes, but only the audio source, headphones 112, and main speaker104 are examples of ports.

Route-Based Policy Feedback

The audio routing manager 212 may furthermore interpret the policyfindings of the audio policy module 208, to ensure that certain policyfindings make sense. For example, the audio routing manager 212 mayreport back to the audio policy module 208 that certain policy findingscannot be implemented. When policy findings can be implemented, theaudio routing manager 212 may then decide how to route the audio throughthe appropriate audio signal processing hardware, as shown in FIG. 4.For example, audio for the headphones 112 may be processed by differentaudio processors than audio routed to the main speaker 104.

Broadcasting Audiovisual Information

Another feature of the audio routing manager 212 is to “broadcast”certain audio (e.g. play the same ring-tone out of main speaker 104,headphones 112, and line out 120). A policy may exist in which it isdesired to play through all speakers, due to desire to not miss atelephone call. This may be suitable in an instance where, for example,the headphones 112 are plugged into the device 100, but not on theuser's ears. Thus, the audio routing module also has provisions for a“broadcast route”—that is, play through multiple speakers at once.

Audio Mixing

According to an embodiment, the media server 204 by itself does notallow changing of routing for existing audio, but may request that theaudio routing manager 212 mix in other audio with the current category,such as UI audio or some type of notification, with the audio that iscurrently playing. Such behavior may be convenient because the mediaserver 204 can make this request without checking with the audio routingmanager to ensure that a route exists.

Example Operation of the Audio Routing Manager

According to an embodiment, the audio routing manager may be used in thefollowing example scenario. Suppose that a user connects a wirelessheadset to the device 100. The media server 204, the audio policymanager 208, and the audio routing manager 212 would “find” the wirelessdevice, and connect it to both uplink (source) and downlink (playback)paths. In this example it will be assumed that the user wishes to hearmusic through the wireless headset, although that may not always be thecase.

Now suppose the user receives a telephone call. The audio policy module208 would first decide how to route the ring tone, and then how to routethe telephone audio. After obtaining the policy findings from the audiopolicy module 208, the audio routing manager 212 would in turn determinehow to route first the ring tone and then the telephone audio.

This process may be achieved as follows. The audio policy module 208 mayfirst check one of the data tables 210 _(1-n), and learn that thewireless headset 116 is connected, but that the telephone receiverspeaker 108 also is connected. The audio policy module 208 may thencheck a priority table, and determines that the wireless headset 116 isa higher priority than the telephone receiver speaker 108. The audiopolicy module 208 would then forward these logical findings to the audiorouting manager 212, which would then select a route for the audio andfacilitate that route.

When the route is complete, the audio routing manager 212 may indicateto the media server 204 a “ready” signal, signifying that a route isavailable and ready for use. The media server 204 could then accept asoftware application's request to play audio. However, in someembodiments, the software application would be configured to understandthat such acceptance is always temporary, and subject to variouschanges.

3.7. Ports, Connections, Nodes, and Routes

As explained above, according to an embodiment, when audio needs to berouted within device 100, a category for the audio may be determined.The manner of routing the audio within device 100 may be selected based,at least in part, on the category thus determined.

For each category, routing information may be stored within a databasethat includes data tables 210 _(1-n). In one embodiment, the routinginformation for a category is organized into one or more routes, andeach route is organized into one or more connections. The followingdefinitions apply herein:

-   -   a port is a logical or physical interface through which audio or        video may be routed and to which various input or output        components may be connected (e.g. a mic-in port, line-out port,        serial port, or Bluetooth port);    -   a connection is a path from an audio beginning point to an audio        end point (e.g. a path from an audio-in port to a speaker), and        is a means by which two or more nodes are connected;    -   a node is any component involved in a connection, including        ports, input or output components, audio processors, software        applications, and so on; and    -   a route is a group of one or more connections.

FIG. 5 shows an example grammar for routes, connections, and nodes,according to one embodiment. The arrangement of FIG. 5 is for examplepurposes only. Within FIG. 5, two audio categories are shown: ringtoneand music. To simplify the example, each of the categories includes asingle route. However, each category may include many routes. In oneembodiment, the routes for a category are prioritized within the datatables 210 _(1-n). These prioritized lists that belong to a category arecollectively referred to as the priority-ranked route list of thecategory. From FIG. 5, it is apparent that a single route can containmultiple connections, and that a single connection contains two or morenodes.

Other embodiments may accept requests to route audiovisual informationwith metadata that indicates characteristics of the audiovisualinformation besides or in addition to the categories. For example, themetadata may indicate a sampling frequency or encoding mechanism. In anembodiment, a request may be made with no metadata at all. For example,policy, routing, and connection data may include data for a defaultcategory instead of or in addition to other categories, and the defaultcategory may be used for all requests unless the request includescategory information.

3.8. Routing Status Data

According to an embodiment, a resource manager such as system 200 maymaintain status data for each of a number of input or output components.For example, the “present?” column of table 210 ₁ in FIG. 3A may beconsidered connection status data, in that it indicates that a componentis connected to the device.

FIG. 8 depicts an example table 810 for storing policy and status data,according to an embodiment. Table 810 is another example of a possibletable 210 _(1-n). As with table 210 ₁, table 810 includes status datafor five different components. Each component corresponds to a row oftable 810. Status data for the five components is maintained in columns812 and 814. The data in column 812 indicates whether or not thecomponent is currently connected. The data in column 814 is consideredrouting status data and indicates whether or not the component isconsidered routable—i.e. whether or not the system is permitted to routeaudiovisual information to or from the resource. The status data may beconsidered a subset of policy data, in that the status data expressespolicies that are specifically conditioned upon the status of thecomponents that may be connected to the device.

In managing routes, the resource manager may access one of or both theconnection status data and the routing status data. For example, table810 indicates that wireless component 116 is connected but not routable.A policy may indicate that audiovisual information should not be routedthrough non-routable components. After consulting the routing statusdata, the resource manager may thus refuse to route audiovisualinformation through wireless component 116. At the same time, aninterface may query the resource manager for information indicating alist of which input or output resources are connected to the device. Forthe purpose of answering this request, the resource manager may relyupon connection status data instead of routing status data, and thusindicate that wireless component 116 is in the list of connectedresources.

Table 810 further includes policy data in columns 816 and 818. Thepolicy data in column 816 indicates policies for how or if the deviceshould modify the routing status data in the event of a connection eventfor the component. For example, main speaker 104 and UI speaker 124 areindicated as inherently routable. The resource manager therefore neednot take action in response to a connection event for these components(of course, these components, having a built-in connection, may nevertrigger a “connection” event). Headphones 112 and line-in/line-outcomponent 120 are indicated as automatically routable, and the resourcemanager may therefore automatically set their routing status data toindicate that the components are routable upon connection. Wirelesscomponent 116 is indicated as “optionally routable,” and the resourcemanager may therefore automatically set the routing status data toindicate that the components are not routable upon connection. Or, inembodiments where routing status data is assumed to already benon-routable for newly established connections, the resource manager mayimplement a policy of simply not modifying the routing status data foroptionally routable components. Optionally routable components maysubsequently become routable in response to various inputs, as shall besubsequently discussed. In some cases, other policies may override acomponents' “optionally routable” status, causing the resource managerto indicate that certain or all components indicated to be optionallyroutable should be instead made routable upon connection.

The resource manager may consult policy data in column 818 to determinevarious policies for contexts in which audiovisual information may berouted to or from the various components. For example, the resourcemanager may have been informed that a certain stream is a “ring tone,”and thus use the policy data in column 818 to determine a policy forwhich components are allowed to output a “ring tone.” Alternatively oradditionally, table 810 may contain policy data for categories ofaudiovisual information, as in table 210 ₁.

According to an embodiment, a user configuration interface may accessdata in table 810 to determine a list of components that are optionallyroutable or automatically routable. The interface may then indicate tothe user the routing status of each of the listed components. Theinterface may also present controls for setting the routing status for acomponent to “routable” or “not routable.”

Once modified by the user, the routing status for a component may remainso modified for a certain duration of time. For example, a wirelesscomponent may remain routable until its connection to the device islost. The connection may be lost, for instance, due to the componentbeing turned off, the device being turned off, the component being movedout of range, and so on. Alternatively, the wireless component mayremain routable for the duration of a specific use (e.g. a telephonecall), or until the user once again modifies the routing status. Thisduration may be set to a default value for certain types of devices. Theduration may also or instead be controlled by policies and/or userpreferences.

For example, a user may prefer to playback music on a particularwireless headset. The headset may be initially treated as anoptionally-routable device. Therefore, when the wireless connection isestablished with the headset, the headset may be treated as notroutable. The user may then be presented the option of selecting theheadset for playback of music. In response to selecting the headset, theheadset is made routable. The user may then be presented with the optionof indicating that the headset is “always routable”. If the user doesnot make the headset always routable, then the headset is automaticallymade not routable again when the headset is disconnected. However, ifthe user makes the headset always routable, then the headset will remainroutable when disconnected. Because the headset remains routable, theuser would not need to expressly select the headset for music playbackthe next time the headset is reconnected.

As with any policy data, the device may present various interfaces forconfiguring the policy data in columns 816 and 818 and/or the policydata may be set to default values for certain types of components. Forexample, the resource manager may be coded to automatically store policydata in column 816 indicating “optionally routable” for all types ofwireless components and “automatically routable” for all types of wiredcomponents. The device may also present configuration interfaces whereina user may change these policy settings for individual components orgroups of components.

As with any of tables 210 _(1-n), although the data in table 810 isdepicted as being maintained in table form, the data stored therein mayin fact be stored in any suitable form, including text, XML, or binary.Moreover, some or all of the data in table 810 (and tables 210 _(1-n))may be represented differently from embodiment to embodiment. Forexample, the word “yes” might just as easily be enumerated as an integeror Boolean variable. In some embodiments, some or all the data may bebits and organized in such a manner that some or all of the policydeterminations may be made by performing relatively simple operationssuch as masking or bitwise arithmetic.

Predictive Routing Calculations

According to an embodiment, a device may utilize connection status dataand routing status data internally for many purposes. For example,connection status data and routing status data may be utilized to runvarious predictive scenarios to determine how audio would be routed inthe event that the routing status of a connected component suddenlychanged. Such scenarios may be useful, for instance, in computing backuproutes or in indicating to a user a specific component through whichaudiovisual information would be routed should another component beturned off.

For example, a handheld device may have limited real estate in which topresent a resource management interface, and may therefore wish to onlypresent management controls for a select number of audiovisualcomponents. The device may present a first control for a wirelesscomponent, which is used to configure the routing status for the highestpriority wireless component. Below the first control, the device maypresent a second control corresponding to the component through whichaudiovisual information will be routed when the wireless component isnot routable, as indicated by the predictive calculations.

For example, assume that the device has several connected componentscapable of playing back music. Assume that some of these components areoptionally routable, some are automatically routable, and some areinherently routable. When presenting the user with selection options, itmay be undesirable to present the user with a list all of thecomponents. Instead, the user may be presented with a list that includesonly (a) the component that would be selected by the policies if all thecomponents were routable, (b) the component that would be selected bythe policies if all of the optionally routable components were notroutable, and (c) the component that would be selected by the policiesif only the inherently routable components were routable.

The foregoing example is merely one example of how the routabilitycharacteristic of a component may be used in determining what selectionoptions are presented to a user. Many other uses are possible. Forexample, the user may be presented the option of selecting any of theoptionally and automatically routable components, but of the inherentlyroutable components may only be presented with the one that wouldactually be selected by the policies if no optionally or automaticallyroutable components were connected.

Example Process Flow for Utilizing Routing Status

FIG. 9 is an example process flow 900 for routing audiovisualinformation in an electronic device using routing status, according toan embodiment. Though process flow 900 is depicted as having certainelements occurring in a certain order, other embodiments of theinvention may involve similar process flows having more or fewerelements occurring in orders other than that depicted in FIG. 9.

At block 910, a resource manager, such as system 200, executing on adevice, such as system 100, maintains status data for each component ofa plurality of components. The components are each connectable to aninput port of the device, an output port of the device, or to both aninput port and an output port of the device. For example, the componentsmay include components 104, 112, 116, 120, or 124 of device 100. Thestatus data includes a) connection status data indicating whether thecomponent is connected to the device; and b) routing status dataindicating whether the device is permitted to route audiovisualinformation through a port of the device to which the component isconnected.

At block 920, the resource manager further maintains policy dataindicating one or more policies for selecting ports through which thedevice should route audiovisual information. The policy data may includeany data capable of expressing (or being interpreted as expressing)policies such as those described throughout this document. The policydata may include, for instance, various columns within data tables 210_(1-n). The policies may be conditioned upon the status data stored inblock 910. For example, one policy may be to route telephone callsthrough any device that is routable. The policies may further be basedupon other status data for (or properties of) the connected components,the device, and/or software executing at the device. The policies mayfurther be based upon characteristics of the audiovisual informationbeing presented or received—for example categories or other metadatasent with a request to route the audiovisual information. Such metadatamay, for instance, indicate that an audio stream is sampled at a certainfrequency or encoded in a certain format. Policies may then be evaluatedto determine an appropriate playback component.

At block 930, based at least on (a) the connection status data for oneor more components, (b) the routing status data for the one or morecomponents, and (c) the policy data, the resource manager selects one ormore ports for routing audiovisual information through the device. Forexample, the resource manager may evaluate one or more policiesconditioned upon the status data to select one or more components. Theselection of the one or more components may then implicate one or moreports to which the selected one or more components are connected throughtechniques such as described elsewhere in this application. Theevaluation of the policies may further involve consideration of anyother data upon which policies are based, such as explained with respectto block 920. For example, the resource manager may select the “last in”routable component, per the “last in wins” policy.

At block 940, the audiovisual information is routed through the deviceusing the selected one or more ports. In an embodiment, this routing mayentail the resource manager identifying the selected port(s) orcomponent(s) and directing a software application to route the requestedaudiovisual information to or from the selected port(s) or component(s).In an embodiment, this routing may entail the resource manager providinga route for the software application to use in routing the requestedaudiovisual information. In an embodiment, this routing may entail thesoftware application actually routing the audiovisual informationthrough the resource manager itself, which then assumes responsibilityfor correctly routing the audiovisual information to or from theselected port(s) or component(s).

Routing Status Upon Connection

FIG. 10 is an example process flow 1000 for handling routing status forresources that connect to an electronic device, according to anembodiment. Though process flow 1000 is depicted as having certainelements occurring in a certain order, other embodiments of theinvention may involve similar process flows having more or fewerelements occurring in other orders than that depicted in FIG. 10.

At block 1010, a resource manager for the device detects that an inputor output component has connected to the device. For example, a wire mayhave been plugged into a jack of the device. As another example, awireless accessory device may have negotiated one or more wirelessconnections to the device through various protocols. In response todetecting the connection, the resource manager may set connection statusdata for the component to indicate that the component is connected.

At block 1020, the resource manager determines if the component isautomatically routable. In some cases—for example, the first time acomponent is connected to the device—this determination may be ahard-coded default determination, which may or may not be based on thetype or identity of the device. In an embodiment, this determination maybe based on policy data, such as maintained in column 816 of table 810.Such policy data may have been configured, for instance, by a hardwarevendor, operating system vendor, or user. The policy data may bespecific to the component or applicable to certain types of components.In an embodiment, the resource manager further bases the determinationon 1) whether or not the device has previously received user inputindicating that the device is permitted to route audiovisual informationto or from the component, along with 2) data indicating whether or notthat input is still valid (i.e. the permission was not expressly orimplicitly revoked due to a lost connection or user action).

At block 1030, in response to the resource manager determining that thecomponent is automatically routable, the resource manager sets therouting status data for the component to indicate that the device ispermitted to route audiovisual information to or from the component.

Conversely, at block 1040, in response to the resource managerdetermining that the component is not automatically routable, theresource manager does not modify the routing status data to indicatethat the component is routable. In some embodiments, this block mayentail the resource manager setting the routing status data to indicatethat the component is not routable. In other embodiments, this block mayentail the resource manager doing nothing with the routing status data,as the resource manager assumes that the routing status data would havebeen made “not routable” in step 1080 below if the device were no longerpermitted to route data to or from the component.

After block 1040, then, the routing status data for the component willindicate that the device is not permitted to route audiovisualinformation to or from the component. The process flow may proceed intwo different directions from this point. First, the flow may proceed toblock 1050, at which block the device detects input indicating that thedevice is now permitted to route audiovisual information through thecomponent. Such input may result from, for example, the user pressing abutton on the device or the component, the user selecting a control in atouch screen interface for managing input or output resources, or asoftware application on the device issuing a command. Data indicatinghow long such input remains valid (e.g. instructions to automaticallypermit the component to route audio or visual information in subsequentconnections, or instructions defining the “terminal event” of step 1070)may further be stored in accordance with instructions from a user or asdictated by one or more policies. The input of block 1050 may also haveoccurred prior to the connection of block 1010—for example, the user mayhave provided such input during a previous connection along withimplicit or explicit instructions to remember that input for subsequentconnections. In response to this input, the flow may proceed to block1030.

On the other hand, after block 1040, the resource manager may, at block1060, receive a request to determine a route for certain audiovisualinformation. Block 1060 may also be performed at any time after block1030. The resource manager may receive this request, for example, fromsoftware executing at the device.

At block 1062, the resource manager may consult the connection statusdata and the routing status data to determine which components areconnected and routable. At block 1064, if the routing status dataindicates that the device is permitted to route audiovisual informationto or from the component whose connection to the device was detected inblock 1010, then the device routes the audiovisual information throughthe component (assuming no other policies indicate that anothercomponent—for an example, a component connected after block 1010—has ahigher priority). Otherwise, at block 1066, the device routes theaudiovisual information through a component other than the componentwhose connection to the device was detected in block 1010.

In an embodiment, the request of block 1060 may include instructions toexplicitly override any other policies that would normally beconsidered. In view of such override instructions, the component willalways be selected should the flow reach block 1064. In an embodiment,the request of 1060 may include override instructions for only certainpolicies.

From block 1064 or 1066, flow may return to block 1060. Furthermore,from block 1066, flow may also proceed to block 1050, so that thecomponent finally becomes routable. Finally, from block 1064 or 1066,flow may proceed to block 1070.

At block 1070, the device may detect a terminating event indicating thatthe device is no longer permitted to route audiovisual information to orfrom the component. Terminating events may be hard-coded and/orspecified by device-specific, type-specific, or generic policies.Terminating events may include, for instance, a lost connection, powercycling at the component or the device, or any type of input. In anembodiment, policies for defining terminating events may be evaluatedany time certain events occur, so as to determine whether or not theevent is actually a terminating event.

In response to a terminating event, at block 1080, the resource managermay set the routing status data to indicate that the device is notpermitted to route audiovisual information to or from the component.Additionally, if the terminating event was a terminated connection, theconnection status data may also be set to indicate that the component isno longer connected. From block 1080, flow may then proceed to blocks1010 or 1060.

In an embodiment, a connection may be terminated prior to 1070, at whichpoint the connection status data is set to indicate that the componentis no longer connected. However, the termination of the connection maynot be defined as a terminating event for routing purposes, so that uponsubsequent connections, the component may already be indicated asroutable.

3.9. Grouping Components

According to an embodiment, a resource manager such as system 200 maymaintain “grouping” data indicating logical relationships betweenvarious input or output components connected to a device. A used herein,the term “logical relationships” refers to relationships wherein the useor management of one component has implications on the use or managementof another component.

FIG. 11 depicts an example table 1110 for storing policy and groupingdata, according to an embodiment. Table 1110 is another example of apossible table 210 _(1-n). Table 1110 includes status data for sixdifferent components, including an instantiation of the A2DP bluetoothprofile 1111, an instantiation of the HFP bluetooth profile 1112, abluetooth microphone 1113, a built-in speaker 1114, an externalmicrophone 1115, and an external speaker 1116. As with any of theexample tables described herein, table 1110 may include more or fewerrows than depicted, depending on the number of input or outputcomponents connected to the device. Table 1110 may also include more orfewer columns depending upon the policies and grouping relationshipssupported by the device. For example, table 1110 may include any columnof information from any other example table described herein. Moreover,relationship data may be stored in manners other than that depicted—forexample, a single column may be used to store comma-separated lists ofrelationships. Or, relationships may be identified in table 1110 bygroup or accessory device names instead of a list of other components inthe relationship.

Table 1110 includes columns 1122 and 1124 for maintaining grouping dataindicating logical relationships between devices. In this case, columns1122 and 1124 indicate “sibling” and “partner” relationships,respectively. A sibling relationship identifies a group of componentsthat have complementary purposes, and therefore will likely be utilizedtogether. For example, HFP component 1112 and bluetooth microphone 1113may both be logical components of the same bluetooth headset, andtherefore would most likely be used together. As another example, theexternal microphone 1115 and an external speaker 1116, althoughcomponents of a different accessory device, may be grouped together assiblings to indicate that the components should be used in tandem. Manyother sibling relationships are also possible. For example, a number ofdifferent outputs that are typically used concurrently—either forbroadcasting the same audiovisual information, or for separatelypresenting different categories of audiovisual information—may begrouped together.

A partner relationship identifies a group of components that havemutually exclusive purposes, in that one of the components cannot beused at the same time as the other component. According to anembodiment, partners are components with conflicting purposes at thesame accessory device. For example, HFP profile 1112 and A2DP profile1111 both are used to output audio at the same bluetooth device. Sinceaudio may be output via only one of the profiles at any given time, HFPprofile 1112 and A2DP profile 1111 are considered partners.

The determination to group components may be hard-coded or dictated bypolicy. For example, the input and output components of any accessorydevice that has both an input or output component may automatically begrouped as siblings. Competing output resources at the same accessorydevice may automatically be grouped as partners. The determination togroup components may furthermore be based on certain componentcharacteristics, such as profiles supported by the component. Thedetermination may also be based on data indicating historical usagetrends. Components themselves may announce grouping preferences.Grouping policies may be configured by a user or vendor.

Example Uses of Grouping Data

Grouping data may be used for a variety of purposes, some of which areherein described. One such purpose is to simplify resource managementinterfaces for the user. Rather than present controls for each componentof a group, controls may instead be presented for the entire group.Through such an interface, for example, the resource manager may receivean indication that the device is permitted to route audio through anaccessory device, such as a bluetooth headset. In response to thisindication, the resource manager may consult the grouping data toidentify components of the accessory device. The resource manager maythen store, for each of the identified components of the accessorydevice, data indicating that the device is permitted to routeaudiovisual information through the component. This same strategy may beutilized to control any setting common to the entire group, includingvolume and connection status. Furthermore, the device may exposeinterfaces to applications that allow for sending instructions that areapplicable to a group of components, or for receiving status datacollectively from the group of components.

As another example use of grouping data, policies may be designed toidentify a group for routing audiovisual information instead of anindividual component. For example, one policy may be to always routeaudio through a bluetooth headset whenever the bluetooth headset isconnected. However, the bluetooth headset may feature a number ofgrouped components. Once the decision has been made to route audiothrough the bluetooth headset, other policies may be evaluated toidentify exactly which components of the bluetooth headset should beutilized. For example, priority data for each member of a selectedgroup, such as stored in columns 1126 and 1132-1136, may be consulted toidentify a specific component from that group to or from which to routethe audiovisual information.

As another example use of grouping data, when the resource manager hasalready made a decision to route a first audiovisual stream through afirst component, the resource manager may utilize grouping data todetermine how to concurrently route a second audiovisual stream. Forexample, the device may already be outputting sound for a telephone callto HFP component 1112. The resource manager, in determining how to routeincoming sound for the telephone call, may consult sibling data for theHFP component 1112 to determine from which component it should route theincoming sound. Thus, the resource manager may route sound frombluetooth microphone 1113. In some embodiments, the resource manager maymake this decision even if the policy data would otherwise indicate thatanother input component should have higher priority. Similarly, partnerdata may be consulted to ensure that audiovisual information is notrouted through a component that cannot be utilized simultaneously withanother component already in use.

Overriding “Last in Wins” Priority

As discussed above, table 1110 further includes columns 1126 and1132-1136 for maintaining priority data. The priority data is, in onesense, a subset of the policy data, in that it indicates prioritypolicies in various contexts. Data in column 1126 indicates a “last inwins” hierarchical listing of components, as discussed in previoussections. By this listing, in many cases, the resource manager maydetermine which component(s) currently have priority for routing.

In some embodiments, exceptions to the “last in wins” policy may occurwhen the component with highest priority belongs to a group. If theresource manager determines that the highest priority component belongsto a group, the resource manager will treat the component's group—asopposed to the individual component—as having the highest priority. Theresource manager will then employ various tie-breaking mechanisms todetermine which component in the group should have highest priority. Forexample, the resource manager may consult secondary priorityinformation, such as contained in the category-specific priority rankingcolumns 1132-1136 of table 1110, to determine a component to or fromwhich to route the audiovisual information.

In an embodiment, the resource manager may detect that components a andb can be used to route certain audiovisual information. The resourcemanager may then determine whether component a is related to componentb. If component a is not related to b, then the resource manager mayselect the most recently connected of components a and b. If componentsa and b are related, then the resource manager may choose betweencomponents a and b based on one or more factors other than the relativeconnection times of components a and b.

Overriding a “last in wins” policy may be useful, for example, inembodiments where different components of an accessory device connect atdifferent times. For example, many bluetooth devices establishconnections through different protocols in an unpredictable serialmanner instead of simultaneously. For instance, a bluetooth device mayfirst establish an A2DP connection and then establish a HFP connection.Since the HFP connection would technically be considered the “last in”component, the “last in win” policy would dictate that music should bestreamed through the HFP connection, even though the A2DP connection mayin fact be better suited to play back music. Thus, it is more desirableto treat both the A2DP component and the HFP component as a group ofcomponents that connected simultaneously, and to thus resort to otherpolicies for determining priority.

Example Process Flow for Maintaining Grouping Data

FIG. 12 is an example process flow 1200 for handling routing status forresources that connect to an electronic device, according to anembodiment. Though process flow 1200 is depicted as having certainelements occurring in a certain order, other embodiments of theinvention may involve similar process flows having more or fewerelements occurring in other orders than that depicted in FIG. 12.

At block 1210, a first component connects to the device.

At block 1220, a resource manager executing on the device determinesthat the first component is logically related to one or more othercomponents that are or have previously connected to the device. Block1220 may occur as part of an evaluation of hard-coded or data-drivenpolicies. The resource manager may routinely perform this evaluation atcertain intervals or in response to the connection of any component.Moreover, this evaluation may have been performed previous to block 1210while the first component and the one or more other components werepreviously connected to the device.

At block 1230, in response to determining that the first component islogically related to the one or more other components, the resourcemanager stores grouping data specifying a relationship between the firstcomponent and the one or more other components. This data may be storedin, for example, a database table such as table 1110, or in any otherform.

At block 1240, based at least on the grouping data, the resource managerselects one or more ports for routing the audiovisual informationthrough the device. In an embodiment, this block may entail evaluatingany number of policies conditioned upon the grouping data, such as thepriority policies. In an embodiment, this block may entail receivinginput from a user that specifies configuration settings for a certaingroup (as opposed to individual components of a group) indicated in thegrouping data, which settings in turn affect policies for identifyingroutes.

4.0. Implementation Examples 4.1. Example Audio Routing Infrastructure

FIG. 6 shows an example audio infrastructure 600, according to anembodiment. Although data tables 210 _(1-n) may contain varioussuggested routes for a specific category, these suggested routes remainin abstract (uncommitted) format until the port (node) manager 608 canverify that the various routes, connections and nodes are trulyavailable.

Accordingly, the audio infrastructure 600 works as follows. The audiorouting manager 212 receives notification that an audio event hasoccurred and determines the category for the event. The audio routingmanager reviews a prioritized list of routes associated with thatcategory, Such a list may be held, for example, in the data tables 210_(1-n). As shown in FIG. 6, the routing manager sends an abstract routeto the port manager. The port manager 608 then returns a concrete(verified) route to the audio routing manager 212. The audio routingmanager 212 then passes the route to the routing handler 612, whichimplements the route.

For a given category, numerous audio routes may be contained within datatables 210 _(1-n). However, these routes are abstract (potentiallyworkable but not hardware-verified) until the routes can be verified andaffirmed by a combination of the port manager 608 and routing handler612. Abstract routes contain identifiers describing generic node types(e.g. speaker, headphones, wireless headset, etc.) whereas concreteroutes contain objects describing particular nodes, e.g. Acme Bluetoothheadset Model XYZ.

Thus, the inclusion of a route within the data tables 210 _(1-n), doesnot necessarily imply that at a given time that specific route isusable. The status of any particular route may change depending on usageof the device 100. Port manager 608 is therefore utilized to determinewhether a route is truly available (i.e. hardware-verified or concrete),and then pass a concrete route back to the audio routing manager 212.The audio routing manager 212 may also or instead consult data tables210 _(1-n) and need not rely solely on the port manager 608.

There may be circumstances in which a route, connection, and/or nodeappears to audio routing manager 212 to be available, but in fact is notavailable. One such circumstance may be a hardware failure. Accordingly,the audio routing manager 212 may depend on the port manager 608 androuting handler 612 to verify that a route can in fact be implemented.

Where a route is found to be concrete, the port manager 608 instantiatesan object that corresponds to the ports identified in the abstractroutes. The routing manager 212 passes that concrete route to therouting handler 612. The routing handler 612 then makes calls to theobjects that were instantiated by port manager 608, and in so doingcommunicates directly with ports within the device 100 to see whetherthe proposed route connections are in fact achievable. If the answer isthat the ports in the current route are useable, the routing handler 612establishes the specified route or routes, mutes all other routes, andreturns a “success” back to the routing manager 212. If the answer is‘no’, the routing handler 612 returns data indicating a failure, andaudio routing manager 212 attempts to evaluate the next route in thepriority-ranked list.

As stated, the audio routing manager 212 may sometimes be mistaken aboutwhether a route is truly achievable. For example, the device 100 may bebumped or dropped, a hardware error may occur, or the user may make asudden configuration change. Rather than force the audio routing manager212 to track all this ever-changing status of hardware within the device100, this information may be verified real-time on an as-needed basis bythe port manager 608 and routing handler 612. Thus, in an embodiment,the actual availability of specific nodes within the device 100 isdecoupled from the knowledge of the audio routing manager 212, so as toensure that routing manager 212 is not responsible for awareness ofavailability of specific nodes.

Acting on route properties does not occur until the routing handler 612determines that the route can be established. Once the route isestablished, the routing handler 612 then configures the route asindicated by various properties. This may include, for example, settingup volume levels, volume curves, and mixing levels, and so on.

Port manager 608 may inform the routing manager that no concrete routesare available. All three of routing manager 212, port manager 608, androuting handler 612 are free to consult database 210 and make use of thevarious properties contained therein.

Returning to FIG. 4, for a particular route, all connections may bestored in the data tables 210 _(1-n). After the above determination andverification of a concrete route, the various unwanted connections aremuted, but the routes still exist. At a code level, storing and managingthe status of all audio intermediaries increases the amount ofprocessing and complicates the data tables 210 _(1-n). Thus, in someembodiments it may be easier to mask the unwanted routes than tocontinually rebuild/reconstruct new routes every time an event occurs.However, other embodiments may continually rebuild/reconstruct newroutes every time an event occurs.

Example Operation of Routing Infrastructure

As stated in previous sections, the data tables 2101-n may hold lists ofroutes for specific categories, ranked in order of route priority. FIG.7 shows ranked routing priority lists for two example audio categories,according to an embodiment. The categories include “audio soundtrack ofvideo,” and “voice telephone call.” When an event of either of thesecategories occurs, the audio routing manager 212 cycles through thevarious routes associated with that event, as stored in the data tables2101-n. Using the first example from FIG. 6, the audio routing manager212 will select route 1, and pass that route 1 to the port manager 608in an abstract format, as shown in FIG. 6. The audio routing manager 212may be informed by either the port manager 608 or routing handler 612that route 1 is not available. A route may be unavailable for a varietyof reasons, one of which is that a connection or node within that routeis occupied with some other task, or is failing. Supposing route 1 isunavailable, the audio routing manager 212 would then cycle to route 2,for example, and then re-inquire to the port manager 608 and the routinghandler 612.

4.2. Properties

According to an embodiment, various types of metadata associated withroutes, connection, and nodes may also be stored within data tables 210_(1-n) as route properties, connection properties, and node properties,respectively. Changing the properties of a specific route may result inchanges to the properties of all connections and nodes within thatroute, but only to the extent those connections and nodes participate inthat route. Those same connections and nodes may have differentproperties for when they participate in a different route. Returning toFIG. 5, a speaker identifier is shown, having an automatic gain control(AGC) property. The property associated with the speaker identifierindicates that when that particular connection is used, a particularamount of automatic gain control should be applied to the speaker. FIG.5 also shows numerous examples of ellipses signifying that connectionsand routes may also have properties.

A property is data that indicates something about an entity, where theentity is a node, connection, or route. Property data may, for example,indicate parameters for an audio effect, or information that affectswhether the entity is mandatory. The following is a list of exampleproperties. The list is by no means exhaustive.

A first property may indicate a per-connection mixing leveling, whichallows a specification of fixed gain levels to be applied prior to thepoint at which multiple connections are mixed. One example effect ofthis property may occur when a user is participating in a voicetelephone call while playing music. The per-connection mix levelproperty may be used to attenuate the music but leave the call volumeunaltered.

Another property may indicate per-node, per-connection, or per-routevolume caps. Potential purposes for such a property may includehearing-protection for a user or volume balancing to keep volumesconsistent across various output ports.

Another property may indicate per-node, per-connection, or per-routevolume curve specifications associated with connection mix levels.System 200 may direct specific audio processors to drop the volume of aparticular audio signal before that signal reaches any further nodes. Iftwo connections are at different gain levels, one or the other may beleveled before the two connections are mixed, perhaps by the routingmanager 212, although some other component may also perform this task.Volume slider interfaces for allowing a user to change this property maybe linear, logarithmic, exponential, or on some other scale. Such afeature may be useful for tweaking volume curves at a fine level ofcontrol to enable hardware compensation.

Another property may indicate per-node, per-connection, or per-routedigital signal processing (DSP) effects, such as equalization (EQ),dynamic range compression (DRC), automatic gain control (AGC). Theseeffects may be applied selectively. This property may be valuable if,for example, device 100 uses speakers or microphones with limited rangesor uneven frequency responses.

Another property may indicate per-connection optionality for broadcastroutes. As noted earlier, the routing manager 212 passes abstract routesto the port manager 608, which determines whether various nodes areproperly connected. If a route is designated as optional within the datatables 210 _(1-n), and part of that route is not present or available,that route is muted, and never becomes concrete. If a route property isoptional, when the audio routing manager 212 cycles through the variousroutes, it checks the condition upon which the designation ofoptionality is based. If the condition fails, the audio routing manager212 de-activates the route and does not further consider it. Forexample, the system 200 may want to broadcast a ringtone to severalplaces. Suppose that a policy exists to broadcast only high qualityringtones. In such a case, system 200 would only consider those portsthat provide high sample rates, so that connections for those portscould be considered optional under the condition that they supports aminimum sample rate.

Another property may indicate per-port, per-connection, or per-routehints about what to do when encountering specifically identified audioprocessors. For example, if an audio processor has a weak dynamic range(e.g. −10 db to 0 db), a route incorporating that audio processor maycontain a “hint” disqualifying that audio processor from being used in acertain route where outstanding dynamic range is required. Such hintsare stored in the database 210 _(1-n). A hint is a type of property usedto help make a routing decision in the face of ambiguity. For example,if there are two bluetooth headsets available, a hint may exist tochoose the one that has a larger dynamic range, so the hint would be“large dynamic range preferred”. These hints can be evaluated forexample by the port manager 608.

Another property may be an index of an audio processor's propertieswithin the data tables 210 _(1-n), so as to allow determination of morecomplex behavior within code or scripting languages. For example, datatables 210 _(1-n) may store an identifier in the form of a characterstring, which is used to specifically identify an audio processor. Thisfeature may not be exposed to users. It may possible to identify aparticular port and distinguish it from other ports of the same type.For example, there might be three bluetooth headsets paired with a phonethat each provide a unique identifier (UID). The data tables 210 _(1-n)can store information that is specific to a particular port via UID, anduse that information drive policy decisions. One example may be to keeptrack of how reliable a particular port is. If a particular BlueToothheadset is known to frequently drop calls, that headset can be marked as“bad” within the data tables 210 _(1-n), and cause the system 200 topresent a dialog to the user that there may be a problem with thatparticular headset.

Another property may indicate user configurations stored directly indata tables 210 _(1-n). For example, a user may have odd or unusualpreferences counter-intuitive to or not anticipated by the designers ofthe system 200.

Another property may indicate per-connection or per-route echocancellation/noise reduction settings. In addition to facilitating echocancellation/noise reduction, this property may be used, for example, indecisions by the audio routing manager 212 allow or disallow a route”.

Another property may indicate per-port, per-connection, or per-routepreconditions and postconditions, such as minimum dynamic range, ornecessity of volume control. For example, a particular port may requirea minimum dynamic range. If the specified minimum dynamic range cannotbe reached, the audio routing manager 212 will “fail” the route orcompensate somehow, perhaps by forcing other audio processors to adjust.This behavior may ensure, for example, that the nodes and connectionssufficiently satisfy a given set of criteria (conditions).

Another property may indicate per-category prevention of screen dimmingor auto lock. Such a property may be useful for video or gamecategories. A specific video-related route may have a property indicateswhether or not it supports screen-dimming or auto-locking. This propertymay then be used to determine whether or not screen-dimming orauto-locking policies may be applied to the route.

4.3. Aliasing

System 200 may also accommodate per-category aliasing of route lists. Asdiscussed in previous sections, route lists are priority-ranked in datatables 210 _(1-n). If a route list is already known and established—inother words, if all of the work and effort of setting up a route listhas already been performed—that route-list may be named and referencedby an alias. Using the name of an existing route list avoids the need toduplicate data, because each route list may have numerous propertiesassociated therewith. Also, if it is necessary to change some propertiesof a single route-list, all routes with that alias may be effectivelychanged at once just by pointing the alias to another route. Aliasingmay be performed on a per-category, per-port, per-connection, per-node,per device, or per route basis.

4.4. Fall Back

Another property may indicate per-category route list fallback logic,which is used if a specific category somehow fails unexpectedly. System200 may instead fall back to some other category. For example, an audiostream may be in a “music” category. The system may cycle through allavailable routes in the route list for that category, as suggested byFIG. 6. If all routes in a specific category fail, then none of theroutes for that category develop even into abstract format. The system200 will note that something is clearly wrong within device 100. But, onaccount to a fallback preference indicated by the fallback property, thesystem 200 may still recover by resorting to playback in a differentcategory, such as “audio soundtrack”. The system 200 would then cyclethe routes in the fallback category.

5.0. Implementation Mechanism Hardware Overview

According to one embodiment, the techniques described herein areimplemented by one or more special-purpose computing devices. Thespecial-purpose computing devices may be hard-wired to perform thetechniques, or may include digital electronic devices such as one ormore application-specific integrated circuits (ASICs) or fieldprogrammable gate arrays (FPGAs) that are persistently programmed toperform the techniques, or may include one or more general purposehardware processors programmed to perform the techniques pursuant toprogram instructions in firmware, memory, other storage, or acombination. Such special-purpose computing devices may also combinecustom hard-wired logic, ASICs, or FPGAs with custom programming toaccomplish the techniques. The special-purpose computing devices may bedesktop computer systems, portable computer systems, handheld devices,networking devices or any other device that incorporates hard-wiredand/or program logic to implement the techniques.

For example, FIG. 13 is a block diagram that illustrates a computersystem 1300 upon which an embodiment may be implemented. Computer system1300 includes a bus 1302 or other communication mechanism forcommunicating information, and a hardware processor 1304 coupled withbus 1302 for processing information. Hardware processor 1304 may be, forexample, a general purpose microprocessor.

Computer system 1300 also includes a main memory 1306, such as a randomaccess memory (RAM) or other dynamic storage device, coupled to bus 1302for storing information and instructions to be executed by processor1304. Main memory 1306 also may be used for storing temporary variablesor other intermediate information during execution of instructions to beexecuted by processor 1304. Such instructions, when stored in storagemedia accessible to processor 1304, render computer system 1300 into aspecial-purpose machine that is customized to perform the operationsspecified in the instructions.

Computer system 1300 further includes a read only memory (ROM) 1308 orother static storage device coupled to bus 1302 for storing staticinformation and instructions for processor 1304. A storage device 1310,such as a magnetic disk or optical disk, is provided and coupled to bus1302 for storing information and instructions.

Computer system 1300 may be coupled via bus 1302 to a display 1312, suchas a cathode ray tube (CRT), for displaying information to a computeruser. An input device 1314, including alphanumeric and other keys, iscoupled to bus 1302 for communicating information and command selectionsto processor 1304. Another type of user input device is cursor control1316, such as a mouse, a trackball, or cursor direction keys forcommunicating direction information and command selections to processor1304 and for controlling cursor movement on display 1312. This inputdevice typically has two degrees of freedom in two axes, a first axis(e.g., x) and a second axis (e.g., y), that allows the device to specifypositions in a plane.

Computer system 1300 may implement the techniques described herein usingcustomized hard-wired logic, one or more ASICs or FPGAs, firmware and/orprogram logic which in combination with the computer system causes orprograms computer system 1300 to be a special-purpose machine. Accordingto one embodiment, the techniques herein are performed by computersystem 1300 in response to processor 1304 executing one or moresequences of one or more instructions contained in main memory 1306.Such instructions may be read into main memory 1306 from another storagemedium, such as storage device 1310. Execution of the sequences ofinstructions contained in main memory 1306 causes processor 1304 toperform the process steps described herein. In alternative embodiments,hard-wired circuitry may be used in place of or in combination withsoftware instructions.

The term “storage media” as used herein refers to any media that storedata and/or instructions that cause a machine to operation in a specificfashion. Such storage media may comprise non-volatile media and/orvolatile media. Non-volatile media includes, for example, optical ormagnetic disks, such as storage device 1310. Volatile media includesdynamic memory, such as main memory 1306. Common forms of storage mediainclude, for example, a floppy disk, a flexible disk, hard disk, solidstate drive, magnetic tape, or any other magnetic data storage medium, aCD-ROM, any other optical data storage medium, any physical medium withpatterns of holes, a RAM, a PROM, and EPROM, a FLASH-EPROM, NVRAM, anyother memory chip or cartridge.

Storage media is distinct from but may be used in conjunction withtransmission media. Transmission media participates in transferringinformation between storage media. For example, transmission mediaincludes coaxial cables, copper wire and fiber optics, including thewires that comprise bus 1302. Transmission media can also take the formof acoustic or light waves, such as those generated during radio-waveand infra-red data communications.

Various forms of media may be involved in carrying one or more sequencesof one or more instructions to processor 1304 for execution. Forexample, the instructions may initially be carried on a magnetic disk orsolid state drive of a remote computer. The remote computer can load theinstructions into its dynamic memory and send the instructions over atelephone line using a modem. A modem local to computer system 1300 canreceive the data on the telephone line and use an infra-red transmitterto convert the data to an infra-red signal. An infra-red detector canreceive the data carried in the infra-red signal and appropriatecircuitry can place the data on bus 1302. Bus 1302 carries the data tomain memory 1306, from which processor 1304 retrieves and executes theinstructions. The instructions received by main memory 1306 mayoptionally be stored on storage device 1310 either before or afterexecution by processor 1304.

Computer system 1300 also includes a communication interface 1318coupled to bus 1302. Communication interface 1318 provides a two-waydata communication coupling to a network link 1320 that is connected toa local network 1322. For example, communication interface 1318 may bean integrated services digital network (ISDN) card, cable modem,satellite modem, or a modem to provide a data communication connectionto a corresponding type of telephone line. As another example,communication interface 1318 may be a local area network (LAN) card toprovide a data communication connection to a compatible LAN. Wirelesslinks may also be implemented. In any such implementation, communicationinterface 1318 sends and receives electrical, electromagnetic or opticalsignals that carry digital data streams representing various types ofinformation.

Network link 1320 typically provides data communication through one ormore networks to other data devices. For example, network link 1320 mayprovide a connection through local network 1322 to a host computer 1324or to data equipment operated by an Internet Service Provider (ISP)1326. ISP 1326 in turn provides data communication services through theworld wide packet data communication network now commonly referred to asthe “Internet” 1328. Local network 1322 and Internet 1328 both useelectrical, electromagnetic or optical signals that carry digital datastreams. The signals through the various networks and the signals onnetwork link 1320 and through communication interface 1318, which carrythe digital data to and from computer system 1300, are example forms oftransmission media.

Computer system 1300 can send messages and receive data, includingprogram code, through the network(s), network link 1320 andcommunication interface 1318. In the Internet example, a server 1330might transmit a requested code for an application program throughInternet 1328, ISP 1326, local network 1322 and communication interface1318.

The received code may be executed by processor 1304 as it is received,and/or stored in storage device 1310, or other non-volatile storage forlater execution.

6.0. Extensions and Alternatives

In the foregoing specification, embodiments of the invention have beendescribed with reference to numerous specific details that may vary fromimplementation to implementation. Thus, the sole and exclusive indicatorof what is the invention, and is intended by the applicants to be theinvention, is the set of claims that issue from this application, in thespecific form in which such claims issue, including any subsequentcorrection. Any definitions expressly set forth herein for termscontained in such claims shall govern the meaning of such terms as usedin the claims. Hence, no limitation, element, property, feature,advantage or attribute that is not expressly recited in a claim shouldlimit the scope of such claim in any way. The specification and drawingsare, accordingly, to be regarded in an illustrative rather than arestrictive sense.

1. A method comprising: maintaining, at a device, grouping dataspecifying relationships between a plurality of components through whichthe device routes audio or visual information; determining to routeaudio or visual information through a first component of the pluralityof components; in response to determining to route audio or visualinformation through the first component: reading the grouping data toidentify a relationship between the first component and a secondcomponent of the plurality of components; and based at least on thegrouping data indicating the relationship between the first componentand the second component, the device determining whether to route audioor visual information through the second component while routing audioor visual information through the first component; wherein the method isperformed by the device.
 2. The method of claim 1, further comprising:determining or receiving an indication that the device is permitted toroute audio or visual information to or from the an accessory device; inresponse to determining or receiving an indication that the device ispermitted to route audio or visual information to or from the accessorydevice, consulting the grouping data to identify which components of theplurality of components are components of the accessory device; storing,for each of the identified components of the accessory device, dataindicating that the device is permitted to route audio or visualinformation through the accessory device.
 3. The method of claim 1,further comprising: based on the grouping data indicating therelationship between the first component and the second component,presenting a single interface control for managing a setting common toboth the first component and the second component; in response toselection of the single interface control, configuring both of the firstcomponent and the second component to reflect the selection.
 4. Themethod of claim 1, further comprising: wherein determining to routeaudio or visual information through a first component comprisesdetermining to route particular audio or visual information to or fromthe first component; in response to identifying the relationship betweenthe first component and the second component, based at least on thegrouping data and on priority data for each of the first component andthe second component, determining to instead route the particular audioor visual information to or from the second component.
 5. The method ofclaim 1, further comprising: determining that the first component islogically related to the second component in response to at least oneof: determining that the first component and the one or more othercomponents second component serve mutually exclusive purposes;determining that the first component and the second component are bothcomponents of a single multi-port accessory device; or determining thatthe first component and the one or more other components secondcomponent serve complementary purposes; in response determining that thefirst component is logically related to the second component, storingdata indicating the relationship in the grouping data.
 6. The method ofclaim 1, further comprising: determining that particular audio or visualinformation is to be routed through an accessory device; consulting thegrouping data to identify components of the accessory device; based onthe identified components of the accessory device, determining a toroute the particular audio or visual information through a particularcomponent being selected from the identified components, the particularcomponent being the first component.
 7. The method of claim 6, whereinselecting the particular component is based at least on priority datastored in the device for each of the first component and the secondcomponent.
 8. The method of claim 1, wherein the first component is aninstance of a first profile for an accessory device, wherein the secondcomponent is an instances of a profiles, other than the first profile,for the accessory device.
 9. The method of claim 1, further comprising:based at least on the grouping data indicating that the second componentis a sibling components of the first component, determining to route afirst stream through a first port and a second stream through a secondport, wherein the first stream is routed to a port to which the firstcomponent is connected; wherein the second stream is routed to a port towhich the second component is connected; wherein the second stream isrouted concurrently with the first stream.
 10. The method of claim 9,wherein one of the first stream and the second stream is an inputstream, and wherein the other of the first stream and the second streamis an output stream.
 11. The method of claim 1 wherein: the plurality ofcomponents includes at least a particular component that is built-in tothe device or external to the device, wherein the particular componentis one of a headset, a speaker, a microphone, or a bluetooth device, ora component thereof; wherein each of the first component and the secondcomponent is connectable to the device via a plurality of ports at thedevice; each of the plurality of ports is one of an input port of thedevice, an output port of the device, or both an input port and anoutput port of the device; and each of the plurality of portsfacilitates one of a wired connection, a wireless connection, or abuilt-in connection to a connected component.
 12. One or more storagemedia storing instructions which, when executed by one or more computingdevices, cause performance of the method recited in claim
 1. 13. Asystem comprising at least: one or more processors; a memory, coupled tothe one or more processors, storing instructions for executing one ormore software applications; a plurality of ports to which may beconnected to a plurality of components for inputting or outputting audioor visual information; a resource manager configured to handle requestsfrom the one or more software applications to determine routes forstreams of audiovisual information, each route being through at leastone of the plurality of ports; wherein the resource manager is furtherconfigured to maintain grouping data indicating one or more groups ofcomponents in the plurality of components that are logically related;wherein the resource manager is further configured to perform:determining to route audio or visual information through a first portconnected to a first component of the plurality of components; inresponse to determining to route audio or visual information through thefirst component: reading the grouping data to identify a relationshipbetween the first component and a second component of the plurality ofcomponents; and based at least on the grouping data indicating therelationship between the first component and the second component, thedevice determining whether to route audio or visual information througha second port connected to the second component while routing audio orvisual information through the first component.
 14. The system of claim13, wherein the plurality of components includes at least one of abuilt-in component, a pluggable component, or a wireless component,wherein each of the plurality of ports is either a physical port or alogical port.
 15. The system of claim 13, wherein the resource manageris further configured to: determine that a first component has a partnerrelationship to the second component if the first component and thesecond component serve mutually exclusive purposes; if a first stream isto be routed through the first port, and the first component has saidpartner relationship with the second component, determine not to routeother streams through the second port; determine that the firstcomponent has a sibling relationship to the second component if thefirst component and the second component serve complementary purposes;and if a second stream is to be routed through the first port, and thefirst component has said sibling relationship with the second component,determine to route a complementary stream through the second port, saidcomplementary stream having a purpose that is complementary to a purposeof the second stream.
 16. The system of claim 13, further comprising aninterface for managing aspects of individual components on a groupbasis.
 17. One or more non-transitory storage media storing instructionswhich, when executed by one or more computing devices, cause performanceof the method recited in claim
 2. 18. One or more non-transitory storagemedia storing instructions which, when executed by one or more computingdevices, cause performance of the method recited in claim
 3. 19. One ormore non-transitory storage media storing instructions which, whenexecuted by one or more computing devices, cause performance of themethod recited in claim
 4. 20. One or more non-transitory storage mediastoring instructions which, when executed by one or more computingdevices, cause performance of the method recited in claim
 5. 21. One ormore non-transitory storage media storing instructions which, whenexecuted by one or more computing devices, cause performance of themethod recited in claim
 6. 22. One or more non-transitory storage mediastoring instructions which, when executed by one or more computingdevices, cause performance of the method recited in claim
 7. 23. One ormore non-transitory storage media storing instructions which, whenexecuted by one or more computing devices, cause performance of themethod recited in claim
 8. 24. One or more non-transitory storage mediastoring instructions which, when executed by one or more computingdevices, cause performance of the method recited in claim
 9. 25. One ormore non-transitory storage media storing instructions which, whenexecuted by one or more computing devices, cause performance of themethod recited in claim
 10. 26. One or more non-transitory storage mediastoring instructions which, when executed by one or more computingdevices, cause performance of the method recited in claim
 11. 27. Themethod of claim 1, wherein determining to route audio or visualinformation through a first component comprises determining to route afirst stream of audio or visual information through the first component;wherein determining whether to route audio or visual information throughthe second component while routing audio or visual information throughthe first component comprises, in response to determining to route thefirst stream of audio or visual information through the first component,determining not to route the first stream or a second stream of audio orvisual information through the second component.
 28. The method of claim1, wherein determining to route audio or visual information through afirst component comprises determining to route a first stream of audioor visual information through the first component; wherein determiningwhether to route audio or visual information through the secondcomponent while routing audio or visual information through the firstcomponent comprises, in response to determining to route the firststream of audio or visual information through the first component,determining to route the first stream or a second stream of audio orvisual information through the second component.
 29. One or morenon-transitory storage media storing instructions which, when executedby one or more computing devices, cause performance of the methodrecited in claim
 27. 30. One or more non-transitory storage mediastoring instructions which, when executed by one or more computingdevices, cause performance of the method recited in claim 28.